Infrared and visible light fusion method

ABSTRACT

The present invention provides an infrared and visible light fusion method, and belongs to the field of image processing and computer vision. The present invention adopts a pair of infrared binocular camera and visible light binocular camera to acquire images, relates to the construction of a fusion image pyramid and a significant vision enhancement algorithm, and is an infrared and visible light fusion algorithm using multi-scale transform. The present invention uses the binocular cameras and NVIDIATX2 to construct a high-performance computing platform and to construct a high-performance solving algorithm to obtain a high-quality infrared and visible light fusion image. The present invention constructs an image pyramid by designing a filtering template according to different imaging principles of infrared and visible light cameras, obtains image information at different scales, performs image super-resolution and significant enhancement, and finally achieves real-time performance through GPU acceleration.

TECHNICAL FIELD

The present invention belongs to the field of image processing andcomputer vision, adopts a pair of infrared camera and visible lightcamera to acquire images, relates to the construction of a fusion imagepyramid and a significant vision enhancement algorithm, and is aninfrared and visible light fusion algorithm using multi-scale transform.

BACKGROUND

The binocular stereo vision technology based on visible light band isdeveloped to be relatively mature. Visible light imaging has richcontrast, color and shape information, so the matching informationbetween binocular images can be obtained accurately and quickly so as toobtain scenario depth information. However, visible light band imaginghas defects, and the imaging quality thereof is greatly reduced, forexample, in strong light, fog rain, snow or night, which affects thematching precision. Therefore, the establishment of a color fusionsystem by using the complementarity of different band informationsources is an effective way to produce more credible images in specialenvironments. For example, a visible light band binocular camera and aninfrared band binocular camera are used to form a multi-band stereovision system, and the advantage of not being affected by fog, rain,snow and light of infrared imaging is used to can make up for thedeficiency of visible light band imaging so as to obtain more completeand precise fusion information.

Image fusion is a promising research in the field of image processing.Images formed by two different types of imaging sensors or similarsensors with different focal lengths and exposures can be synthesizedinto a more informative image through the image fusion technology, whichis more suitable for later processing and research. The advantages makeimage fusion widely developed in the fields such as remote sensing,camera or mobile phone imaging, monitoring and reconnaissance, andespecially, the fusion of infrared and visible light images plays a veryimportant role in the military field. In recent years, most of fusionmethods are researched and designed based on the transform domainwithout considering the multi-scale detail information of images,resulting in the loss of details in the fused image, for example, thepublic patent CN208240087U [Chinese], an infrared and visible lightfusion system and image fusion device. The traditional multi-scaletransform is mainly composed of linear filters, and easily produces haloartifacts during the decomposition process. Edge keeping filters canavoid the phenomenon of halo artifacts at the edge while betterretaining the edge characteristics of images, and obtain better resultsin image fusion, thus attracting more and more attention. Therefore, thepresent invention realizes the enhancement of details and the removal ofartifacts on the basis of retaining the effective information ofinfrared and visible light images.

SUMMARY

To overcome the defects in the prior art, the present invention providesan infrared and visible light real-time fusion algorithm based onmulti-scale pyramid transform, which constructs an image pyramid bydesigning a filtering template, obtains image information at differentscales, performs image super-resolution and significant enhancement, andfinally achieves real-time performance through GPU acceleration.

The present invention has the following specific technical solution:

An infrared and visible light fusion method, comprises the followingsteps:

1) Converting the color space of a visible light image from an RGB imageto an HSV image, extracting the value information of the color image asthe input of image fusion, and retaining the original hue andsaturation;

2) Creating a filtering kernel template: constructing the correspondingfiltering kernel template according to the sizes of infrared and visiblelight images; for example, when infrared and visible light images of640*480 are input, the kernel template is as follows:

$\quad\begin{bmatrix}0.015 & 0.031 & 0.062 & 0.031 & 0.015 \\0.031 & 0.062 & 0.125 & 0.062 & 0.031 \\0.062 & 0.125 & 0.25 & 0.125 & 0.062 \\0.031 & 0.062 & 0.125 & 0.062 & 0.031 \\0.015 & 0.031 & 0.062 & 0.031 & 0.015\end{bmatrix}$

3) Constructing an image pyramid by using the filtering kernel templateconvolution;

4) Extracting the details of the pyramid at different scales by thelinear interpolation method;

5) Distinguishing the details of infrared and visible light of eachlayer of the pyramid by using the saliency of the details, convolvingthe images by using the designed sliding window to generate a weightmatrix, and comparing the neighborhood information of each pixelaccording to the weight matrix to distinguish and extract more credibledetail images.

6) Linearly adding the infrared and visible light images of the smallestscale to obtain a smooth background image, fusing the extracted detailimages into the background image, performing super-resolution upscalingon the image, then adding the detail information of the upper-layerscale, and iterating successively up to the top of the pyramid.

6-1) The super-resolution technology uses the cubic convolutioninterpolation to obtain richer details of a magnified image than thebilinear interpolation, the weight of each pixel value is determined bythe distance from the pixel to the pixel to be determined, and thedistance includes distances in horizontal and vertical directions.

7) Converting the color space: converting the fusion image back to theRGB image, and adding the hue and saturation previously retained.

8) Enhancing the color: Enhancing the color of the fusion image togenerate a fusion image with higher resolution and contrast. Performingpixel-level image enhancement for the contrast of each pixel.

The present invention has the following beneficial effect: the presentinvention designs a real-time fusion method using infrared and visiblelight binocular stereo cameras. The present invention solves the detailsof infrared and visible light by using the multi-scale pyramidtransform, carries out inverse pyramid transform with thesuper-resolution technology, constructs a highly credible fusion image,and has the following characteristics: (1) the system is easy toconstruct, and the input data can be acquired by using stereo binocularcameras; (2) the program is simple and easy to implement; (3) betterdetails of the image are obtained by pyramid multi-scale transform; (4)the problem of loss of inverse pyramid transform details is effectivelymade up with the super-resolution technology; (5) the structure iscomplete, multi-thread operation can be performed, and the program isrobust; and (6) the detail images are used to perform significantenhancement and differentiation to improve the generalization ability ofthe algorithm.

DESCRIPTION OF DRAWINGS

FIG. 1 shows infrared and visible light images acquired by anacquisition platform and a schematic diagram of pyramid multi-scaledecomposition.

FIG. 2 is a step chart of overall image acquisition and fusion.

FIG. 3 is a flow chart of the present invention.

FIG. 4 is a final fusion image of the present invention.

DETAILED DESCRIPTION

The present invention proposes a method for real-time image fusion by aninfrared camera and a visible light camera, and will be described indetail below in combination with drawings and embodiments.

A binocular stereo camera is placed on a fixed platform. In theembodiment, the image resolution of the camera is 1280×720, and thefield angle is 45.4°; and the experimental platform is shown in FIG. 1 ,and NVIDIATX2 is used for calculation to ensure timeliness. On thisbasis, an infrared and visible light fusion method is provided, and themethod comprises the following steps:

1) Obtaining registered infrared and visible light images

1-1) Respectively calibrating each lens of the visible light binocularcamera and the infrared binocular camera and jointly calibrating therespective systems;

1-2) Respectively calibrating the infrared binocular camera and thevisible light binocular camera by the Zhangzhengyou calibration methodto obtain internal parameters such as focal length and principal pointposition and external parameters such as rotation and translation ofeach camera;

1-3) Calculating the positional relationship of the same plane in thevisible light image and the infrared image by using the externalparameter RT obtained by the joint calibration method and the detectedchecker corners, and registering the visible light image to the infraredimage by using a homography matrix.

2) Performing multi-scale pyramid transform on the images, and using thedesigned filtering template to respectively perform down-convolution anddown-sampling on the infrared image and the visible light image, asshown in FIG. 1 , wherein the filtering template acquisition mode isshown in the formula below:

$\begin{matrix}{{h\left( {x,y} \right)} = e^{- \frac{x^{2} + y^{2}}{2\sigma^{2}}}} & (1)\end{matrix}$wherein x is the distance between other pixels and the center pixel inthe neighborhood; y is the distance between other pixels and the centerpixel in the neighborhood; and a is a standard deviation parameter.

3) Extracting the details of the infrared and visible light images basedon multi-scale pyramid transform, and using the high frequency of theimages obtained by the linear interpolation method as the detail layerof fusion.

The following is the main flow of the algorithm as shown in FIG. 2 , andthe specific description is as follows:

4) Converting the color space of the image

4-1) In view of the problem that the visible light image has RGB threechannels, converting the RGB color space to the HSV color space,extracting the V (value) information of the visible light image to befused with the infrared image, and retaining H (hue) and S (saturation),wherein the specific conversion is shown as follows:

$\begin{matrix}{R^{\prime} = \frac{R}{255}} & (2) \\{G^{\prime} = \frac{G}{255}} & (3) \\{B^{\prime} = \frac{B}{255}} & (4) \\{{C\;\max} = {\max\mspace{11mu}\left( {R^{\prime},G^{\prime},B^{\prime}} \right)}} & (5) \\{{C\;\min} = {\min\mspace{11mu}\left( {R^{\prime},G^{\prime},B^{\prime}} \right)}} & (6) \\{\Delta = {{C\;\max} - {C\;\min}}} & (7) \\{V = {C\;\max}} & (8)\end{matrix}$wherein R is a red channel, G is a green channel, and B is a bluechannel; R′ is the red channel after color space conversion, G′ is thegreen channel after color space conversion, and B′ is the blue channelafter color space conversion; Cmax represents the maximum value amongR′, G′, B′; Cmin represents the minimum value among R′, G′, B′; and Δrepresents the difference between the maximum value and the minimumvalue among R′, G′, B′;

4-2) Extracting the V (value) channel as the input of visible light,retaining the hue H and saturation S to the corresponding matrix, andretaining the color information for the subsequent color restorationafter fusion.

5) Convolving details by filtering, and filtering infrared and visiblelight detail images

5-1) Designing two 3×3 empty matrixes, starting convolution sequentiallyfrom the starting pixels of the two images, distinguishing eightneighborhood pixels of the corresponding points in the visible light andinfrared detail images, distinguishing the saliency stationary points ofthe corresponding neighborhood pixels, taking 1 for large ones and 0 forsmall ones, and respectively saving in the corresponding matrixes; andupdating sequentially till the last pixel of the image;

5-2) According to the weight of the generated matrix, fusing the detailimages of the infrared and visible light images to generate a detailimage with richer texture.

6) Performing inverse multi-scale pyramid transform by using imagesuper-resolution

6-1) Selecting the cubic convolution interpolation super-resolutionalgorithm for inverse multi-scale pyramid transform; from the deepestdown-sampled sub-image, after fusing the detail images, expanding theimage to the second deepest sub-image by super-resolution, and iteratingsuccessively until restoring to the original image size; with a pixel asan example, the distances between the pixel and the pixel to bedetermined in the vertical and horizontal directions are respectively1+u and v, the weight of the pixel is w=w(1+u)×w(v), and then the pixelvalue f(i+u,j+v) of the pixel to be determined is calculated as follows:f(i+u,j+v)=A×Q×P  (9)wherein A, Q and P are matrixes generated by the distances; andA=[w(1+u) w(u) w(1−u) w(2−u)];

$\mspace{20mu}{{P = \begin{bmatrix}{w\left( {1 + v} \right)} & {{w(v)}\ } & {\ {w\left( {1 - v} \right)}} & {\ {w\left( {2 - v} \right)}}\end{bmatrix}^{T}};}$ $Q = {\begin{bmatrix}{f\left( {{i - 1},{j - 1}} \right)} & {f\left( {{i - 1},{j + 0}} \right)} & {f\left( {{i - 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 0},{j - 1}} \right)} & {f\left( {{i + 0},{j + 0}} \right)} & {f\left( {{i + 0},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 1},{j - 1}} \right)} & {f\left( {{i + 1},{j + 0}} \right)} & {f\left( {{i + 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 2},{j - 1}} \right)} & {f\left( {{i + 2},{j + 0}} \right)} & {f\left( {{i + 2},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)}\end{bmatrix}.}$

The interpolation kernel w(x) is:

$\begin{matrix}{{w =}\left\{ \begin{matrix}{1 - {2{x}^{2}} + {x}^{3}} & {{x} < 1} \\{4 - {8{x}} + {5{x}^{2}} - {x}^{3}} & {1 \leq {x} < 2} \\0 & {{x} \geq 2}\end{matrix} \right.} & (10)\end{matrix}$Finally, according to the weight and value of the pixel, calculating thepixel value of the corresponding position of the pixel aftersuper-resolution;

6-2) Saving the super-resolved fusion image in a newly established zeromatrix to prepare for the next step.

7) Performing restoration and color enhancement on the color space

7-1) Restoring to the RGB color space from the HSV color space by savingthe super-resolved fusion image into the V (value) channel for updatingin combination with the previously retained H (hue) and S (saturation),wherein the specific update formulas are shown as follows:

$\begin{matrix}{C = {V \times S}} & (11) \\{X = {C \times \left( {1 - {{{\left( {H\text{/}60{^\circ}} \right)\mspace{11mu}{mod}\mspace{11mu} 2} - 1}}} \right)}} & (12) \\{m = {V - C}} & (13) \\{\left( {R^{\prime},G^{\prime},B^{\prime}} \right) = \left\{ \begin{matrix}{\left( {C,X,0} \right),} & {{0{^\circ}} \leq H < {60{^\circ}}} \\{\left( {X,C,0} \right),} & {{60{^\circ}} \leq H < {120{^\circ}}} \\{\left( {0,C,X} \right),} & {{120{^\circ}} \leq H < {180{^\circ}}} \\{\left( {0,X,C} \right),} & {{180{^\circ}} \leq H < {240{^\circ}}} \\{\left( {X,0,C} \right),} & {{240{^\circ}} \leq H < {300{^\circ}}} \\{\left( {C,0,X} \right),} & {{300{^\circ}} \leq H < {360{^\circ}}}\end{matrix} \right.} & (14) \\{R^{\prime},G^{\prime},{B^{\prime} = \left( {{\left( {R^{\prime} + m} \right) \times 255},{\left( {G^{\prime} + m} \right) \times 255},{\left( {B^{\prime} + m} \right) \times 255}} \right)}} & \left( {15} \right)\end{matrix}$wherein C is the product of the value and the saturation; and m is thedifference between the value and C.

7-2) Performing color correction and enhancement on the image restoredin step 7-1) to generate a three-channel image that is more consistentwith observation and detection; and performing color enhancement on theR channel, G channel and B channel respectively, as shown in theformulas below:

$\begin{matrix}{R_{out} = \left( R_{in} \right)^{{1/g}amma}} & (16) \\{R_{display} = \left( R_{in}^{({{1/g}amma})} \right)^{gamma}} & (17) \\{G_{out} = \left( G_{in} \right)^{{1/g}amma}} & (18) \\\left. {G_{display} = \left( G_{in} \right)^{({{1/g}amma})}} \right)^{gamma} & (19) \\{B_{out} = \left( B_{in} \right)^{{1/g}amma}} & (20) \\{B_{display} = \left( B_{in}^{({{1/g}amma})} \right)^{gamma}} & (21)\end{matrix}$wherein gamma is a brightness enhancement parameter; R_(out) is inversetransform of the red channel after gamma correction; R_(in) is the valueof the initial red channel; R_(display) is the value of the R channelafter gamma correction; G_(display) is the value of the G channel aftergamma correction; and B_(display) is the numerical compensation value ofthe B channel after gamma correction. The generated image is shown inFIG. 4 .

The invention claimed is:
 1. An infrared and visible light fusionmethod, wherein the fusion method comprises the following steps: 1)obtaining registered infrared images and visible light images; 2)performing multi-scale pyramid transform on the infrared images andvisible light images, and using the designed filtering template torespectively perform down-convolution and down-sampling on the infraredimages and the visible light images, wherein the filtering templateacquisition mode is: $\begin{matrix}{{h\left( {x,y} \right)} = e^{- \frac{x^{2} + y^{2}}{2\sigma^{2}}}} & (1)\end{matrix}$ wherein x is a horizontal distance between other pixelsand the center pixel in neighborhood; y is a vertical distance betweenother pixels and the center pixel in the neighborhood; and a is astandard deviation parameter; 3) extracting the details of the infraredimages and visible light images based on multi-scale pyramid transform,and using the high frequency of the infrared images and visible lightimages obtained by the linear interpolation method as the detail layerof fusion; 4) converting a color space of the image infrared images andvisible light images; 5) convolving details by filtering, and filteringinfrared detail images and visible light detail images; 6) performinginverse multi-scale pyramid transform by using image super-resolution;7) performing restoration and color enhancement on the color space;wherein step 1) comprises the following specific steps: 1-1)respectively calibrating each lens of the visible light binocular cameraand the infrared binocular camera and jointly calibrating the respectivesystems; 1-2) respectively calibrating the infrared binocular camera andthe visible light binocular camera by the Zhangzhengyou calibrationmethod to obtain internal parameters and external parameters of eachcamera, wherein the internal parameters include focal length andprincipal point position, and the external parameters include rotationand translation; 1-3) calculating the positional relationship of thesame plane in the visible light images and the infrared images by usingthe external parameter RT obtained by the joint calibration method andthe detected checker corners, and registering the visible light imagesto the infrared images by using a homography matrix.
 2. The infrared andvisible light fusion method according to claim 1, wherein step 4)comprises the following specific steps: 4-1) when the visible lightimages have RGB three channels, converting an RGB color space to an HSVcolor space, wherein the specific conversion is shown as follows:$\begin{matrix}{R^{\prime} = \frac{R}{255}} & (2) \\{G^{\prime} = \frac{G}{255}} & (3) \\{B^{\prime} = \frac{B}{255}} & (4) \\{{C\;\max} = {\max\mspace{11mu}\left( {R^{\prime},G^{\prime},B^{\prime}} \right)}} & (5) \\{{C\;\min} = {\min\mspace{11mu}\left( {R^{\prime},G^{\prime},B^{\prime}} \right)}} & (6) \\{\Delta = {{C\;\max} - {C\;\min}}} & (7) \\{V = {C\;\max}} & (8)\end{matrix}$ wherein R is a red channel, G is a green channel, and B isa blue channel; R′ is the red channel after color space conversion, G′is the green channel after color space conversion, and B′ is the bluechannel after color space conversion; Cmax represents the maximum valueamong R′, G′, B′; Cmin represents the minimum value among R′, G′, B′;and Δ represents the difference between the maximum value and theminimum value among R′, G′, B′; 4-2) extracting value information V asan input of visible light, retaining hue H and saturation S to thecorresponding matrix, and retaining the color information for thesubsequent color restoration after fusion.
 3. The infrared and visiblelight fusion method according to claim 1, wherein step 5) comprises thefollowing specific steps: 5-1) designing two 3×3 empty matrixes,starting convolution sequentially from the starting pixels of thevisible light detailed images and the infrared detail images,distinguishing eight neighborhood pixels of the corresponding points inthe visible light detailed images and the infrared detail images,distinguishing the saliency stationary points of the correspondingneighborhood pixels, taking 1 for large ones and 0 for small ones, andrespectively saving in the corresponding matrixes; and updatingsequentially till the last pixel of the image; 5-2) according to aweight of the generated matrix, fusing the visible light detailed imagesand the infrared detail images of the infrared images and visible lightimages to generate a detail image with rich texture.
 4. The infrared andvisible light fusion method according to claim 2, wherein step 5)comprises the following specific steps: 5-1) designing two 3×3 emptymatrixes, starting convolution sequentially from the starting pixels ofthe two images, distinguishing eight neighborhood pixels of thecorresponding points in the visible light detail images and the infrareddetail images, distinguishing the saliency stationary points of thecorresponding neighborhood pixels, taking 1 for large ones and 0 forsmall ones, and respectively saving in the corresponding matrixes; andupdating sequentially till the last pixel of the image; 5-2) accordingto the weight of the generated matrix, fusing the detail images of theinfrared images and visible light images to generate a detail image withrich texture.
 5. The infrared and visible light fusion method accordingto claim 1, wherein step 6) comprises the following specific steps: 6-1)selecting a cubic convolution interpolation super-resolution algorithmfor inverse multi-scale pyramid transform; from a deepest down-sampledsub-image, after fusing the infrared detail images and the visible lightdetail images, expanding the deepest down-sampled sub-image to a seconddeepest sub-image by super-resolution, and iterating successively untilrestoring to the original image size; in the case of a pixel, thedistances between the pixel and the pixel to be determined in thevertical and horizontal directions are respectively 1+u and v, theweight of the pixel is w=w(1+u)×w(v), and then the pixel value f(i+u,j+v) of the pixel to be determined is calculated as follows:f(i+u,j+v)=A×Q×P  (9) wherein A, Q and P are matrixes generated by thedistances; and A=[w(1+u) w(u) w(1−u) w(2−u)];$\mspace{20mu}{{P = \begin{bmatrix}{w\left( {1 + v} \right)} & {{w(v)}\ } & {\ {w\left( {1 - v} \right)}} & {\ {w\left( {2 - v} \right)}}\end{bmatrix}^{T}};}$ ${Q = \begin{bmatrix}{f\left( {{i - 1},{j - 1}} \right)} & {f\left( {{i - 1},{j + 0}} \right)} & {f\left( {{i - 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 0},{j - 1}} \right)} & {f\left( {{i + 0},{j + 0}} \right)} & {f\left( {{i + 0},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 1},{j - 1}} \right)} & {f\left( {{i + 1},{j + 0}} \right)} & {f\left( {{i + 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 2},{j - 1}} \right)} & {f\left( {{i + 2},{j + 0}} \right)} & {f\left( {{i + 2},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)}\end{bmatrix}};$ the interpolation kernel w(x) is: $\begin{matrix}{{w =}\left\{ \begin{matrix}{1 - {2{x}^{2}} + {x}^{3}} & {{x} < 1} \\{4 - {8{x}} + {5{x}^{2}} - {x}^{3}} & {1 \leq {x} < 2} \\0 & {{x} \geq 2}\end{matrix} \right.} & (10)\end{matrix}$ finally, according to the weight and value of the pixel,calculating the pixel value of the corresponding position of the pixelafter super-resolution; 6-2) saving the super-resolved fusion image in anewly established zero matrix.
 6. The infrared and visible light fusionmethod according to claim 3, wherein step 6) comprises the followingspecific steps: 6-1) selecting a cubic convolution interpolationsuper-resolution algorithm for inverse multi-scale pyramid transform;from a deepest down-sampled sub-image, after fusing the infrared detailimages and the visible light detail images, expanding the deepestdown-sampled sub-image to a second deepest sub-image bysuper-resolution, and iterating successively until restoring to theoriginal image size; in the case of a pixel, the distances between thepixel and the pixel to be determined in the vertical and horizontaldirections are respectively 1+u and v, the weight of the pixel isw=w(1+u)×w(v), and then the pixel value f (i+u,j+v) of the pixel to bedetermined is calculated as follows:f(i+u,j+v)=A×Q×P  (9) wherein A, Q and P are matrixes generated by thedistances; and A=[w(1+u) w(u) w(1−u) w(2−u)];$\mspace{20mu}{{P = \begin{bmatrix}{w\left( {1 + v} \right)} & {{w(v)}\ } & {\ {w\left( {1 - v} \right)}} & {\ {w\left( {2 - v} \right)}}\end{bmatrix}^{T}};}$ ${Q = \begin{bmatrix}{f\left( {{i - 1},{j - 1}} \right)} & {f\left( {{i - 1},{j + 0}} \right)} & {f\left( {{i - 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 0},{j - 1}} \right)} & {f\left( {{i + 0},{j + 0}} \right)} & {f\left( {{i + 0},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 1},{j - 1}} \right)} & {f\left( {{i + 1},{j + 0}} \right)} & {f\left( {{i + 1},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)} \\{f\left( {{i + 2},{j - 1}} \right)} & {f\left( {{i + 2},{j + 0}} \right)} & {f\left( {{i + 2},{j + 1}} \right)} & {f\left( {{i + 2},{j + 2}} \right)}\end{bmatrix}};$ the interpolation kernel w(x) is: $\begin{matrix}{{w =}\left\{ \begin{matrix}{1 - {2{x}^{2}} + {x}^{3}} & {{x} < 1} \\{4 - {8{x}} + {5{x}^{2}} - {x}^{3}} & {1 \leq {x} < 2} \\0 & {{x} \geq 2}\end{matrix} \right.} & (10)\end{matrix}$ finally, according to the weight and value of the pixel,calculating the pixel value of the corresponding position of the pixelafter super-resolution; 6-2) saving the super-resolved fusion image in anewly established zero matrix.
 7. The infrared and visible light fusionmethod according to claim 1, wherein step 7) comprises the followingspecific steps: 7-1) restoring to an RGB color space from an HSV colorspace by saving a super-resolved fusion image into the value informationV for updating in combination with the previously retained H (hue) and S(saturation), wherein the specific formulas are shown as follows:$\begin{matrix}{C = {V \times S}} & (11) \\{X = {C \times \left( {1 - {{{\left( {H\text{/}60{^\circ}} \right)\mspace{11mu}{mod}\mspace{11mu} 2} - 1}}} \right)}} & (12) \\{m = {V - C}} & (13) \\{\left( {R^{\prime},G^{\prime},B^{\prime}} \right) = \left\{ \begin{matrix}{\left( {C,X,0} \right),} & {{0{^\circ}} \leq H < {60{^\circ}}} \\{\left( {X,C,0} \right),} & {{60{^\circ}} \leq H < {120{^\circ}}} \\{\left( {0,C,X} \right),} & {{120{^\circ}} \leq H < {180{^\circ}}} \\{\left( {0,X,C} \right),} & {{180{^\circ}} \leq H < {240{^\circ}}} \\{\left( {X,0,C} \right),} & {{240{^\circ}} \leq H < {300{^\circ}}} \\{\left( {C,0,X} \right),} & {{300{^\circ}} \leq H < {360{^\circ}}}\end{matrix} \right.} & (14) \\{R^{\prime},G^{\prime},{B^{\prime} = \left( {{\left( {R^{\prime} + m} \right) \times 255},{\left( {G^{\prime} + m} \right) \times 255},{\left( {B^{\prime} + m} \right) \times 255}} \right)}} & \left( {15} \right)\end{matrix}$ wherein C is a product of the value and the saturation;and m is a difference between the value and C; 7-2) performing colorcorrection and enhancement on the super-resolved fusion image restoredin step 7-1) to generate a three-channel image that is consistent withobservation and detection; and performing color enhancement on an Rchannel, a G channel and a B channel respectively, wherein the specificformulas are shown as follows: $\begin{matrix}{R_{out} = \left( R_{in} \right)^{{1/g}amma}} & (16) \\{R_{display} = \left( R_{in}^{({{1/g}amma})} \right)^{gamma}} & (17) \\{G_{out} = \left( G_{in} \right)^{{1/g}amma}} & (18) \\\left. {G_{display} = \left( G_{in} \right)^{({{1/g}amma})}} \right)^{gamma} & (19) \\{B_{out} = \left( B_{in} \right)^{{1/g}amma}} & (20) \\{B_{display} = \left( B_{in}^{({{1/g}amma})} \right)^{gamma}} & (21)\end{matrix}$ wherein gamma is a brightness enhancement parameter;R_(out) is inverse transform of the red channel after gamma correction;R_(in) is the value of the initial red channel; R_(display) is the valueof the R channel after gamma correction; G_(display) is the value of theG channel after gamma correction; and B_(display) is the numericalcompensation value of the B channel after gamma correction.